Systems and methods for use at a vehicle including an eye tracking device

ABSTRACT

Systems and methods for a vehicle including an eye tracking device. The systems and methods use input from the eye tracking device. The systems and methods are configured to communicate with a driver based on input from the eye tracking device. For example, the systems and methods are configured to generate indicators on a display based on input from the eye tracking device.

TECHNICAL FIELD

The present disclosure relates generally to systems and methods for useat a vehicle including an eye tracking device.

BACKGROUND

Conventional in-vehicle user interfaces and instrument clusters includecomplex displays having multiple visual outputs presented thereon. Thesedisplays require a relatively high amount of driver attention and, insome cases, the use of hands to interact with the user interfaces. Suchdisplays could distract a driver resulting in less safe drivingconditions.

SUMMARY

The present technology relates to systems and methods including an eyetracking device.

According to an exemplary embodiment, a method includes displaying anindicator on a display of a vehicle. The indicator includes an indicatorarea and is associated with a vehicle system. The method furtherincludes comparing sensor data to a threshold. The sensor data isassociated with the vehicle system. The threshold represents aseparation between a first state or operation, e.g., a normal state, ofthe vehicle system and a second state or operation, e.g., a criticalstate, of the vehicle system. The method further includes comparing gazelocation data to the indicator area; and reducing a prominence of theindicator if: the sensor data is on a normal state side of thethreshold; and the gaze location data is found in the indicator area.

According to an exemplary embodiment, a method includes displaying anindicator on a display of a vehicle. The indicator includes an indicatorarea and is associated with a vehicle system. The method furtherincludes comparing sensor data to a threshold. The sensor data isassociated with the vehicle system. The threshold represents aseparation between a normal state of the vehicle system and a criticalstate of the vehicle system. The method further includes comparing gazelocation data to the indicator area; and increasing a prominence of theindicator if: sensor data is on a critical state side of the threshold;and gaze location data is not found in the indicator area.

According to an exemplary embodiment, a method includes receivinginformation to be communicated to a driver; and displaying anotification indicator on a display. The notification indicator includesan indicator area. The method further includes comparing gaze locationdata to the indicator area; and communicating the information if thegaze location data is found in the indicator area.

According to an exemplary embodiment, a method includes displaying afirst indicator on a vehicle display. The first indicator includes afirst indicator area. The method further includes calculating a firstgaze frequency associated with the first indicator. The first gazefrequency is calculated over a first period of time and is based on oneof: a number of times a gaze location moves into the first indicatorarea; and a time a gaze location spends in the first indicator area. Themethod further includes determining a prominence of the first indicatorbased on the first gaze frequency.

According to an exemplary embodiment, the method further includesdisplaying a second indicator on a vehicle display. The second indicatorincludes a second indicator area. The method further includescalculating a second gaze frequency associated with the secondindicator. The second gaze frequency is calculated over a second periodof time and is based on one of: a number of times a gaze location movesinto the second indicator area; and a time a gaze location spends in thesecond indicator area. The method further includes determining theprominence of the second indicator based on the second gaze frequency.Positions on the display are ordered based on prominence. Determiningthe prominence includes determining the position of the first indicatorand the second indicator based on an order of the first gaze frequencyand the second gaze frequency.

According to an exemplary embodiment, a method includes displaying eachof a first indicator and a second indicator on a display of a vehicle.The first indicator includes a first indicator area and the secondindicator includes a second indicator area. A first distance separatesthe first indicator and the second indicator. The method furtherincludes analyzing gaze location data on the display; and decreasing thefirst distance if a gaze pattern between the first indicator and thesecond indicator is identified. The gaze pattern is based on a number oftransitions between the first indicator and the second indicator.

According to an exemplary embodiment, a method includes accessing a setof default parameter values; and comparing the set of default parametervalues to output data from a sensor. The sensor is associated with avehicle system of a vehicle. The method further includes communicating,if the output data matches one of the set of default parameter values,default information associated with the one of the set of defaultparameter values; registering a gaze associated with the vehicle systemif gaze location data is found at a location associated with the vehiclesystem; determining, at a time when the gaze is registered, an auxiliaryparameter; generating auxiliary information based on the auxiliaryparameter; and communicating the auxiliary information.

According to an exemplary embodiment, a method includes registering agaze associated with a vehicle system if gaze location data is found ata location associated with the vehicle system; and communicating, at atime when the gaze is registered, information associated with thevehicle system.

The method further includes determining a context based on sensor data;and determining the information associated with the vehicle system basedon the context.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a partial view of a vehicle, according to anembodiment of the present disclosure.

FIG. 2 illustrates a schematic illustration of a display of the vehicleof FIG. 1.

FIG. 3 illustrates a computing device of the vehicle of FIG. 1.

FIG. 4 illustrates a declutter method according to a declutterapplication of the computing device of FIG. 3.

FIGS. 5-7 schematically illustrate the declutter method of FIG. 4.

FIG. 8 illustrates a system notification method according to a systemnotification application of the computing device of FIG. 3.

FIGS. 9-11 schematically illustrate the system notification method ofFIG. 8.

FIG. 12 illustrates a delayed notification method according to a delayednotification application of the computing device of FIG. 3.

FIGS. 13-15 schematically illustrate the delayed notification method ofFIG. 12.

FIG. 16 illustrates a driver request method according to a driverrequest application of the computing device of FIG. 3.

FIGS. 17-20 schematically illustrate the driver request method of FIG.16.

FIG. 21 illustrates an arrangement method according to an arrangementapplication of the computing device of FIG. 3.

FIGS. 22-24 schematically illustrate the arrangement method of FIG. 21.

FIG. 25 illustrates a relationship method according to a relationshipapplication of the computing device of FIG. 3.

FIGS. 26-28 schematically illustrate the relationship method of FIG. 25.

FIG. 29 illustrates an information adjustment method according to aninformation adjustment application of the computing device of FIG. 3.

FIGS. 30-35 schematically illustrate the information adjustment methodof FIG. 29.

FIG. 36 illustrates a vehicle system information method according to avehicle system information application of the computing device of FIG.3.

FIG. 37 schematically illustrates the vehicle system information methodof FIG. 36.

FIG. 38 illustrates a driver context information method according to adriver context information application of the computing device of FIG.3.

FIG. 39 schematically illustrates the driver context information methodof FIG. 38.

The figures are not necessarily to scale and some features may beexaggerated or minimized, such as to show details of particularcomponents. In some instances, well-known components, systems, materialsor methods have not been described in detail in order to avoid obscuringthe present disclosure. Therefore, specific structural and functionaldetails disclosed herein are not to be interpreted as limiting, butmerely as a basis for the claims and as a representative basis forteaching one skilled in the art to variously employ the presentdisclosure.

DETAILED DESCRIPTION

As required, detailed embodiments of the present disclosure aredisclosed herein. The disclosed embodiments are merely examples that maybe embodied in various and alternative forms, and combinations thereof.As used herein, for example, “exemplary,” and similar terms, referexpansively to embodiments that serve as an illustration, specimen,model or pattern.

While the present technology is described primarily herein in connectionwith automobiles, the technology is not limited to automobiles. Theconcepts can be used in a wide variety of applications, such as inconnection with aircraft, marine craft, and other vehicles.

According to an embodiment illustrated in FIG. 1, a vehicle 10 includesvehicle systems 20, controls 22 for the vehicle systems 20, sensors 30,a computing device 40, and displays 60. Certain of the sensors 30 areconfigured to output data 62 reflecting measurements of parameters ofthe vehicle systems 20. Certain of the sensors 30 are configured tooutput data 64 reflecting measurements of an environment 50 of thevehicle 10.

Vehicle systems 20 are also configured to output data 66 or otherwiseprovide another source of information. For example, the output data 66is generated by the vehicle system 20. Vehicle systems 20 that generateoutput data 66 include electronic control units.

Also, information (e.g., information 68) from vehicle system 20 isavailable on a controller area network (CAN) Bus. For example, whetheror not the automatic cruise control (ACC) is engaged, the CAN busincludes information about whether the radio is on and at what volume.The CAN bus also includes personalization information that is availableto all vehicle systems 20 that indicates the identity of the driver andtheir preferences.

The computing device 40 is configured to receive the output data 62, 64,66. The computing device 40 is configured to store the output data 62,64, 66 as information 68 or to generate the information 68 based on theoutput data 62, 64, 66.

The computing device 40 is further configured to communicate with adriver via a vehicle-user interface such as the displays 60, an audio(e.g., speaker/microphone) system 88, or a haptic system (e.g., in thesteering wheel 82). For example, to communicate the information 68visually, the computing device 40 is configured to generate and positionindicators 70 on the displays 60 or generate and position text on thedisplays 60. To communicate the information 68 audibly, the computingdevice 40 is configured to generate an audio signal based on theinformation 68 and play the audio file through the audio system 88.

The display 60 is a visual output. For example, the display 60 can be atwo-dimensional output or a three-dimensional output. Two-dimensionaloutput includes an electronic output on a screen or a projection onto asurface. Three-dimensional outputs include holographic projections.

For purposes of teaching, a device with a display 60 is described indetail below as including an electronic screen on which digital outputis displayed. For example, the indicators 70 are digital images that arepositioned on the display 60 by the computing device 40.

Referring to FIG. 2, the display 60 includes display areas 74 atpositions 72. Here, each of the indicators 70 is associated with one ofthe display areas 74 of the display 60. Each of the indicators 70includes an indicator area 76. For example, the indicator area 76 isless than or equal to the associated display area 74 and is scaled asdescribed in further detail below. Alternatively described, theindicators are targets and the indicator areas are target regions.

The display 60 can be that of an instrument panel, a human-machineinterface (HMI), an entertainment or infotainment system (e.g., radio,video playing systems), a navigation system, a system that connects toauxiliary devices (e.g., bluetooth devices, cellular phones, or anyother system brought into the vehicle—here, information is presented forexample by a smartphone projection or smartphone connection) or theauxiliary device itself, a head up display (HUD) (e.g., that isprojected onto a windshield of the vehicle), other devices for providingvisual communication, and the like.

Referring to FIG. 1, for purposes of teaching, the display 60 ispositioned in a dashboard 80 of the vehicle 10 behind the steering wheel82. In some embodiments, a display includes one or more screens,projections, and the like. For example, a display 60 can include a firstdisplay 60 behind the steering wheel 82, a second display 60 (e.g., aHUD) projected onto the windshield 84 of the vehicle 10, and a thirddisplay 60 in the center of the dashboard 80 (e.g., center console 86).Although a display can include physically separate components, thecomponents can be treated as a single display. Otherwise, the componentscan operate as individual displays or different displays can be formedfrom different components.

In certain embodiments, the indicators 70 represent a state or conditionof the vehicle systems 20 or environment 50. Commonly, indicators 70include gauges such as a speedometer, tachometer, odometer, and fuelgauge. Other indicators 70 include a gearshift position, a seat beltwarning light, a parking-brake-engagement warning light, and anengine-malfunction light, low fuel, low oil pressure, low tire pressure,and faults in the airbag (SRS) system.

Other indicators 70 can relate to vehicle systems 20 such as heatingsystems, air conditioning systems, braking systems, accelerationsystems, entertainment or infotainment systems (e.g., radio, videoplaying systems), navigation systems, mirrors (e.g., mirror adjustmentsystems), seats (e.g., seat adjustment systems), window control systems,doors (e.g., door lock control systems), collision-avoidance systems,traction control systems, a horn, windshield wiper systems, belts andhoses, emission system, engine, engine cooling system, exhaust system,lighting and wipers, starting, charging, and batteries, steering andsuspension, transmission, sensors, switches, HVAC, cameras,communication devices (e.g., OnStar® devices and other wirelesscommunication devices), systems that connect to auxiliary devices (e.g.,Bluetooth devices, cellular phones), cluster, center stack, head updisplay (HUD), speech, gestures, sound, and the like.

Similarly, controls 22 relate to vehicle systems 20.

Continuing with FIGS. 1 and 2, the vehicle 10 further includes a gazedetection device 90 that is configured to output gaze location data 92reflecting a gaze location 94 over time. For example, the gaze location94 is the location of a gaze of a driver on the display 60.

For example, the gaze detection device 90 detects a gaze location 94based on the position of the user's eye, the position of the display 60,and a direction of the eye. Here, the gaze location 94 is where avector, intersecting the position of the user's eye and having the angleof the direction of the user's eye, intersects the plane of the surfaceof the display 60. The position of the user's eye depends on, forexample, a head pose.

According to an exemplary embodiment, the gaze detection device 90includes a camera 96 that is configured to capture an image of at leasta portion of a head (e.g. face or eyes) of the user and to generate asignal representing the image. The gaze detection device 90 alsoincludes a source of radiant energy, such as an infra-red light emittingdiode 98, to illuminate at least a portion of the head of the user. Incertain embodiments, more than one camera is used for eye tracking toimprove the accuracy of the gaze detection device 90.

The gaze detection device 90 is configured to analyze the imagescaptured by the camera 96 to determine the position of the eye and thegaze angle. For example, the gaze detection device 90 includes acomputing device similar to the computing device 40 described below andan analysis application for processing the images.

Alternatively the gaze detection device 90 provides the images to thecomputing device 40 and the computing device 40 includes an analysisapplication to analyze the images.

For purposes of teaching, gaze location data described below isillustrated as gaze locations 94 measured over time (x-axis). Gazelocation data moves along a y-axis to illustrate the movement of thegaze location 94 between various locations in the vehicle 10 (e.g.,locations on the display 60). Areas that include a number of locationsin the vehicle 10 are indicated by a range on the y-axis.

In the figures, time periods of fixation at a certain location areindicated by a flat line. Time periods of transition between locationsin different areas are indicated by a sloped line. Time periods ofsaccades between different locations in the same area are indicated bysloped line that remains in an area.

FIG. 3 illustrates the computing device 40 of the vehicle 10 of FIG. 1.In certain embodiments, the computing device 40 includes an applicationprogramming interface (API) and a user interface (UI) generator. Incertain embodiments, the computing device is or includes that of asmartphone.

The computing device 40 includes a processor 100 for controlling and/orprocessing data, input/output data ports 102, and a memory 110. Theprocessor could be multiple processors, which could include distributedprocessors or parallel processors in a single machine or multiplemachines. The processor could include virtual processor(s). Theprocessor could include a state machine, application specific integratedcircuit (ASIC), programmable gate array (PGA) including a Field PGA, orstate machine. When a processor executes instructions to perform“operations,” this could include the processor performing the operationsdirectly and/or facilitating, directing, or cooperating with anotherdevice or component to perform the operations.

The computing device 40 can include a variety of computer-readablemedia, including volatile media, non-volatile media, removable media,and non-removable media. The term “computer-readable media” and variantsthereof, as used in the specification and claims, includes storagemedia. Storage media includes volatile and/or non-volatile, removableand/or non-removable media, such as, for example, RAM, ROM, EEPROM,flash memory or other memory technology, CDROM, DVD, or other opticaldisk storage, magnetic tape, magnetic disk storage, or other magneticstorage devices or any other medium that is configured to be used tostore information that can be accessed by the computing device 40.

While the memory 110 is illustrated as residing proximate the processor100, it should be understood that at least a portion of the memory canbe a remotely accessed storage system, for example, a server on acommunication network (e.g. a remote server), a remote hard disk drive,a removable storage medium, combinations thereof, and the like. Thus,any of the data, applications, and/or software described below can bestored within the memory and/or accessed via network connections toother data processing systems (not shown) that may include a local areanetwork (LAN), a metropolitan area network (MAN), or a wide area network(WAN), for example.

The memory 110 includes several categories of software and data used inthe computing device 40, including, applications 120, a database 130, anoperating system 140, and input/output device drivers 150.

As will be appreciated by those skilled in the art, the operating system140 may be any operating system for use with a data processing system.The input/output device drivers 150 may include various routinesaccessed through the operating system 140 by the applications tocommunicate with devices, and certain memory components. Theapplications 120 can be stored in the memory 110 and/or in a firmware(not shown) as executable instructions, and can be executed by theprocessor 100.

The applications 120 include various programs that, when executed by theprocessor 100, implement the various features of the computing device40, including declutter applications, system notification applications,delayed notification applications, driver request applications,arrangement applications, relationship applications, informationadjustment applications, vehicle system information applications, anddriver context information applications, each of which is described infurther detail below. The applications 120 are stored in the memory 110and are configured to be executed by the processor 100.

The applications 120 may be applied to data stored in the database 130,such as that of signals received by the sensors 30 (e.g., received viathe input/output data ports 102 along with data received over a wirelessdata connection). The database 130 represents the static and dynamicdata used by the applications 120, the operating system 140, theinput/output device drivers 150 and other software programs that mayreside in the memory 110.

It should be understood that FIG. 3 and the description above areintended to provide a brief, general description of a suitableenvironment in which the various aspects of some embodiments of thepresent disclosure can be implemented. The terminology“computer-readable media”, “computer-readable storage device”, andvariants thereof, as used in the specification and claims, can includestorage media. Storage media can include volatile and/or non-volatile,removable and/or non-removable media, such as, for example, RAM, ROM,EEPROM, flash memory or other memory technology, CDROM, DVD, or otheroptical disk storage, magnetic tape, magnetic disk storage, or othermagnetic storage devices or any other medium, excluding propagatingsignals, that can be used to store information that can be accessed bythe device shown in FIG. 3.

While the description refers to computer-readable instructions,embodiments of the present disclosure also can be implemented incombination with other program modules and/or as a combination ofhardware and software in addition to, or instead of, computer readableinstructions.

While the description includes a general context of computer-executableinstructions, the present disclosure can also be implemented incombination with other program modules and/or as a combination ofhardware and software. The term “application,” or variants thereof, isused expansively herein to include routines, program modules, programs,components, data structures, algorithms, and the like. Applications canbe implemented on various system configurations, includingsingle-processor or multiprocessor systems, minicomputers, mainframecomputers, personal computers, hand-held computing devices,microprocessor-based, programmable consumer electronics, combinationsthereof, and the like.

FIGS. 4-39 illustrate methods and applications according to embodimentsof the present disclosure. Applications includes computer-executableinstructions that, when executed by the processor 100, cause theprocessor 100 to perform the associated method.

It should be understood that the steps of methods are not necessarilypresented in any particular order and that performance of some or allthe steps in an alternative order is possible and is contemplated. Thesteps have been presented in the demonstrated order for ease ofdescription and illustration. Steps can be added, omitted and/orperformed simultaneously without departing from the scope of theappended claims.

It should also be understood that the illustrated methods can be endedat any time. In certain embodiments, some or all steps of this process,and/or substantially equivalent steps are performed by execution ofcomputer-readable instructions stored or included on a computer readablemedium, such as the memory 110 of the computing device 40 describedabove, for example.

Referring to FIGS. 4-7, the declutter application 120 includescomputer-readable instructions that, when executed by the processor 100,cause the processor 100 to perform a declutter method 200.

At a block 210, the processor 100, displays (e.g., facilitates displayof) an indicator 70, as shown in FIG. 5, and analyzes sensor data 212(e.g., horizontal x-axis is time) from the sensor 30—sensor 30 shown inFIG. 1 and sensor data 212 shown in FIG. 6.

Continuing with FIG. 6, if the sensor data 212 is on a normal operationside of (e.g., above or below) a threshold 214, at a block 220, theprocessor 100 analyzes gaze location data (e.g., gaze locations 94measured over time, movement between a gaze location 94 outside theindicator area 76 to a gaze location 94 inside the indicator area 76 isrepresented by arrow 222). For example, the threshold 214 representsseparation between a first operation, e.g., a normal operation and asecond operation, e.g., a critical operation, of a respective one of thevehicle systems 20.

Referring again to FIG. 5, if the gaze location data includes a gazelocation 94 in the indicator area 76 of the indicator 70 that isassociated with the vehicle system 20 of the sensor 30, at a block 230,the processor 100 displays the indicator 70 such that the prominence ofthe indicator 70 is reduced. Reducing the prominence includes reducingsize, brightness, color, font, type of indicator, moving to a lessprominent position (e.g., away from the center of the display),combinations thereof, and the like.

As an example, referring to FIGS. 5-7, the processor 100 displays a fuellevel indicator 70 and analyzes fuel level data 212 from the fuel levelsensor 30. If the fuel level data 212 shows the fuel level is above afuel level threshold 214, the processor 100 analyzes gaze location data.If the fuel level data 212 is greater than the fuel level threshold 214,the fuel level data 212 is acceptable and the information is notimmediately important once the driver is aware of the fuel level. If thegaze location data includes a gaze location 94 in an indicator area 76of the fuel level indicator 70, the driver has noticed the fuel leveland the processor 100 reduces the prominence of the fuel level indicator70 by changing the type of graphic used and/or reducing the size of thegraphic.

Other examples include the method 200 where the engine temperature data212 is measured by an engine temperature sensor 30 and compared to anengine temperature threshold 214 (e.g., the engine temperature threshold214 defines a “yellow zone”); the method 200 where air pressure data 212is measured by an air pressure sensor 30 in one of the wheels andcompared to an air pressure threshold 214 (e.g., the air pressurethreshold is a pressure that is low but not critical such as 28 PSI fora tire whose normal is 32 PSI); and the method 200 where oil life data212 is measured by an oil life sensor 30 and compared to an oil lifethreshold 214 (e.g., the oil life exceeds 100% meaning that an oilchange is needed). In these examples, engine temperature data 212 belowthe engine temperature threshold 214 is normal, air pressure data 212above the air pressure threshold 214 is normal, and oil life data 212below the oil life threshold 214 is normal.

Referring to FIGS. 8-11, the system notification application 120includes computer-readable instructions that, when executed by theprocessor 100, cause the processor 100 to perform the systemnotification method 300.

At a block 310, the processor 100, displays an indicator 70 on thedisplay 60 as shown in FIG. 9 and analyzes data 312 (e.g., horizontalx-axis is time) from a sensor 30 as shown in FIG. 10.

Continuing with FIG. 10, if the sensor data 312 is on a criticaloperation side of (above or below) a threshold 314, at a block 320, theprocessor 100 analyzes gaze location data (e.g., gaze locations 94measured over time, movement between a gaze location 94 outside theindicator area 76 to a gaze location 94 inside the indicator area 76 isrepresented by arrow 322 as shown in FIG. 11). For example, thethreshold 314 represents operation of a vehicle system 20 below which awarning is to occur.

Continuing with FIG. 9, at a block 330, if the gaze location data doesnot include a gaze location 94 in the indicator area 76 of the indicator70, the processor 100 increases the prominence of the indicator 70. Forexample, the prominence of the indicator 70 is increased (e.g., at arate or for a time) until gaze location data includes a gaze location 94in the indicator area 76 of the indicator 70, which is associated withthe vehicle system 20 of the sensor 30. Increasing the prominenceincludes increasing size, brightness, type of indicator, moving to amore prominent position 72 (e.g., toward the center of the display),combinations thereof, and the like.

As an example, referring to FIGS. 9-11, the processor 100 displays afuel level indicator 70 and analyzes data 312 from the fuel level sensor30. If the fuel level sensor data 312 shows the fuel level is below afuel level threshold 314, the fuel level is not acceptable and thedriver is to be notified. The processor 100 analyzes gaze location data.If the gaze location data does not include a gaze location 94 in anindicator area 76 of the fuel level indicator 70, the driver is unawareof the fuel level and the processor 100 increases the prominence of thefuel level indicator 70 by increasing the size of the graphic. Once thegaze location data includes a gaze location 94 in an indicator area 76of the fuel level indicator 70, the driver has noticed the fuel level.

Other examples include the method 300 where the engine temperature data212 is measured by an engine temperature sensor 30 and compared to anengine temperature threshold 214 (e.g., the engine temperature threshold214 defines a “red zone”); the method 300 where air pressure data 212 ismeasured by an air pressure sensor 30 in one of the wheels and comparedto an air pressure threshold 214 (e.g., the air pressure threshold is apressure that is critical); and the method 300 where oil life data 212is measured by an oil life sensor 30 and compared to an oil lifethreshold 214 (e.g., the oil life exceeds 125% meaning that an oilchange is overdue). In these examples, engine temperature data 212 abovethe engine temperature threshold 214 is critical, air pressure data 212below the air pressure threshold 214 is critical, and oil life data 212above the oil life threshold 214 is critical.

As another example, road position data is compared to a lane (e.g.,threshold) and a lane departure warning is communicated to the driverwith haptic feedback until the driver looks at a warning indicator orindicator of a lane departure system.

Referring to FIGS. 12-15, the delayed notification application 120includes computer-readable instructions that, when executed by theprocessor 100, cause the processor 100 to perform a delayed notificationmethod 400.

Referring to FIG. 13, at a block 410, the processor 100 receivesinformation 68 or generates the information 68. The processor 100 thengenerates and displays an indicator 70 that represents an awaitingnotification, and analyzes gaze location data 412.

At a block 420, if the gaze location data (e.g., gaze locations 94measured over time, movement between a gaze location 94 outside theindicator area 76 to a gaze location 94 inside the indicator area 76 isrepresented by arrow 412) includes a gaze location 94 in an indicatorarea 76 of the indicator 70, referring to FIG. 14, the processor 100displays the information 68 (e.g., replaces the indicator 70 with theinformation 68). In alternative embodiments, the gaze location 94 is inthe indicator area 76 for a certain amount of time before theinformation 68 is displayed.

The indicator 70 may be generated and displayed based on data from asensor 30. For example, referring to FIG. 15, at the block 310, theprocessor 100 generates and displays an indicator 70 that represents anawaiting notification only if sensor data 422 (e.g., horizontal x-axisis time) is below a threshold 424. Alternatively or additionally, at theblock 420, the processor 100 displays the information 68 only if sensordata 422 is below the threshold 424.

Here, the threshold 424 represents operation of a vehicle system 20below which (or above which) a driver can receive the information 68without substantially distracting or disturbing the driver. For example,the threshold 424 may be a speed, an amount of traffic, a time to nextdriving direction, etc.

As an example, referring to FIGS. 13-14, the processor 100 receives amessage 68, generates and displays an indicator 70 (e.g., an envelopesymbol) on the display 60, and analyzes gaze location data 412. If thegaze location data 412 includes a gaze location 94 in an indicator area76 of the indicator 70, the processor 100 displays the message 68 on thedisplay 60. In certain embodiments, for example, the processor 100 onlydisplays the indicator 70 if the speed is measured by a speed sensor 30to be below ten miles per hour.

Referring to FIGS. 16-20, the driver request application 120 includescomputer-readable instructions that, when executed by the processor 100,cause the processor 100 to perform the driver request method 500.

At a block 510, referring to FIGS. 17 and 18, the processor 100 displaysthe indicators 70 and analyzes gaze location data 512.

At a block 520, referring to FIGS. 18 and 19, a gaze, or glance,frequency 522 is calculated for each of the indicators 70. The gazefrequency 522 for an indicator 70 is the number of times a gaze location94 moves into the indicator area 76 (e.g., transitions T) of theindicator 70 over a time period 524. In alternative embodiments, thegaze frequency 522 is time a gaze location 94 spends in the indicatorarea 76 (e.g., dwell time equals sum of all fixations and saccadesbetween transitions or shifts of gaze to other target areas) over thetime period 524. Glance duration can be defined as the time from themoment at which the direction of gaze moves toward a target to themoment it moves away from it. A single glance duration may be referredto as a glance.

The gaze frequency can also be defined as the number of glances to atarget within a pre-defined sample time period, or during a pre-definedtask, where each glance is separated by at least one glance to adifferent target.

At a block 530, referring to FIGS. 19 and 20, the prominence of eachindicator 70 is increased or decreased based on the respectivecalculated gaze frequency 522 of the indicator 70. For example, aprominence of each indicator 70 is associated with an average gazefrequency 532. If the calculated gaze frequency 522 is above the averagegaze frequency 532, the processor 100 increases the prominence of theindicator 70. If the calculated gaze frequency 522 is below the averagegaze frequency 532, the processor 100 decreases the prominence of theindicator 70.

In certain embodiments, average gaze frequency 532 is the gaze frequency522 that is calculated over a longer time period.

Referring to FIGS. 21-24, the arrangement application 120 includescomputer-readable instructions that, when executed by the processor 100,cause the processor 100 to perform the arrangement method 600.

According to the arrangement method 600, indicators 70 with greateraverage gaze frequency 532, 534 are displayed at more prominentpositions 72 that have and displayed towards the center of the display60 or at the most prominent position.

Referring to FIGS. 22 and 23, at a block 610, the processor 100 displaysthe indicators 70 based on previously calculated average gaze frequency532 (e.g., long-term average) and analyzes gaze location data 512.Referring momentarily to FIGS. 19 and 23, for each indicator 70, theprocessor 100 calculates a current gaze frequency 522 (e.g., the numberof gaze locations 94 in the indicator area 76 of the indicator 70 or thetime that the gaze location 94 is in the indicator area 76 of theindicator 70 over the last one minute 524) and calculates an updatedaverage gaze frequency 534 based on the current gaze frequency 522.

At a block 620, referring to FIG. 23, the processor 100 analyzes theupdated average gaze frequencies 534 of the indicators 70.

At a block 630, referring to FIG. 24, the processor 100 arranges theindicators 70 in the display areas 74 at the positions 72 based on theupdated average gaze frequencies 534. For example, positions 72 on thedisplay 60 and size ranges at those positions 72 are predefined andranked by prominence (i.e., position 72 and display area 74).

Generally, the indicators 70 with the larger updated average gazefrequencies 534 are arranged in positions 72 with a relatively largersize range (e.g., the size of indicator area 76 within display area 74is determined based on current gaze frequency 522 as described abovewith respect to method 500) at a more central (or easy to find) position72 on the display 60. Similarly, the indicators 70 with the smallerupdated average gaze frequencies 534 are arranged in positions 72 with arelatively smaller size range (e.g., the size of indicator area 76within display area 74 is determined based on current gaze frequency 522as described above with respect to method 500) and a less centralposition 72 on the display 60.

In certain embodiments, a HUD includes a most prominent position 72.

Referring to FIGS. 25-28, the relationship application 120 includescomputer-readable instructions that, when executed by the processor 100,cause the processor 100 to perform the relationship method 700.

At a block 710, referring to FIG. 26, the processor 100 displays theindicators 70 and analyzes gaze location data 712 (e.g., horizontalx-axis is time).

At a block 720, referring to FIG. 27, a gaze pattern 722 is identified.For example, the gaze pattern 722 is identified when a gaze location 94is found in the indicator area 76 of at least two different ones of theindicators 70 in at least two different time periods 724, 726.Alternatively, a gaze pattern 722 is identified when a number oftransitions between two indicators 70 is greater than a threshold numberof transitions.

At a block 730, referring to FIG. 28, if a gaze pattern 722 isidentified, the processor 100 decreases a distance 732 between theindicators 70 associated with the gaze pattern 722 or otherwise combinesthe indicators 70 associated with the gaze pattern 722.

Referring to FIGS. 29-35, the information adjustment application 120includes computer-readable instructions that, when executed by theprocessor 100, cause the processor 100 to perform the informationadjustment method 800. According to the information adjustment method800, default information is augmented with auxiliary information orotherwise altered based on a gaze of a driver.

The information 68 to be communicated by the computing device 40 may bestatic (e.g., fixed or predetermined content already in memory) or maybe dynamic (e.g., may change over time). Dynamic information includes,for example, time of day, weather, and driver state.

The information 68 may be generated and stored to be accessed by thecomputing device 40. Alternatively, the computing device 40 may generatethe information 68 in real-time or request that the vehicle system 20generate the information 68 in real-time. For purposes of teaching,information 68 is categorized below as default information and auxiliaryinformation.

The memory 110 includes set of default parameter values 812. Exemplaryparameters include location, distance, and time.

At a block 810, the processor 100 monitors the output data 62 of asensor 30 or vehicle system 20. The output data 62 includes values of aparameter associated with the set of default parameter values 812. Theprocessor 100 compares the output data 62 of the sensor 30 to the set ofdefault parameter values 812.

When the output data 62 of the sensor 30 matches one of the set ofdefault parameter values 812 (e.g., FIG. 31), at a block 820, theprocessor 100 accesses, generates, or requests a default information 822based on the matched one of the set of default parameter values 812(e.g., FIG. 32). For example, the processor 100 accesses or generatesthe default information 822 associated with the matched one of the setof default parameter values 812; or requests that the defaultinformation 822 is generated by the vehicle system 20 and receives thedefault information 822.

At a block 830, the processor 100 communicates the default information822 to the driver. For example, the communication is visual on thedisplay 60 and/or audible through the audio system 88 of the vehicle 10.

At a block 840, the processor 100 analyzes gaze location data 842 (e.g.,horizontal x-axis is time) to monitor a gaze location 94 of a driver(e.g., FIG. 33). If the gaze location 94 is on the display 60 (e.g.,HMI) for a time 844 that is greater than a threshold time 846, a gaze isregistered and the processor 100 generates an auxiliary parameter value848 at a time 849 the gaze is registered (e.g., FIG. 34). Alternativelydescribed, the auxiliary parameter value 848 modifies (e.g., is addedto) the set of default parameter values 812.

Alternatively, a gaze is registered if a gaze frequency exceeds acertain threshold. Gaze frequency is the number of times a gaze location94 is on the display 60 over a time period as described above withrespect to gaze frequency 522.

At a block 850, the processor 100 generates (or requests generation of)an auxiliary information 852 based on the auxiliary parameter value 848(e.g., FIG. 35).

At a block 860, the auxiliary information 852 is communicated to thedriver. For example, the auxiliary information 852 is communicatedvisually on the display 60 and/or audibly through the audio system 88 ofthe vehicle 10.

In this manner, the computing device 40 responds to the gaze location 94of a driver by increasing the communication of information 68 to thedriver. For example, the frequency of communication of information 68 isincreased by communicating the auxiliary information 852 in addition tothe default information 822.

In certain embodiments, a gaze triggers the use, in place of the set ofdefault parameters values 812, of a new set of parameter values. The newset of parameter values causes more frequent communication to beprovided to the driver as compared to the set of default parametervalues 812. For example, the new set of parameter values is a more fullset of parameter values (or otherwise increases the number of the set ofdefault parameter values 812).

In certain embodiments, if a gaze of a driver is not found in gazelocation data during a period of time, the absence of a gaze reduces thedefault communication to the driver. For example, the absence of a gazetriggers the use, in place of the set of default parameter values 812,of a new set of parameter values. The new set of parameter values causesless frequent communication to be provided to the driver as compared tothe set of default parameter values 812. For example, new set ofparameter values is a more sparse set of parameters (or otherwisereduces the number of the set of default parameter values 812).

Referring to FIGS. 30-35, an example application is now described. Here,the information 68 is a direction that is associated with a parameter(e.g., locations, including locations based on geographical or temporalparameters) along a route 862. The route 862 is generated by thecomputing device 40 or the vehicle system 20, one or both of which is anavigation system.

For purposes of teaching, the parameter is a location. Default locationvalues 812 can be determined in various ways. For example, the defaultlocation values 812 can be based on relative distance to a turn or otherdirection along the route 862; or the time to a turn or other directionalong the route 862 given the relative distance to the turn and thecurrent speed of the vehicle.

The sensor 30 (e.g., of the navigation system 20) is a location sensor30, the output data 62 of which is the vehicle location 62.

Referring to FIG. 30, the processor 100 monitors the vehicle location 62and compares the vehicle location 62 to the set of default locationvalues 812.

Referring to FIGS. 31 and 32, when the vehicle location 62 matches oneof the set of default location values 812, the processor 100 generates adefault direction 822 based on the matched one of the set of defaultlocation values 812. The processor 100 communicates the defaultdirection 822 to the driver.

Referring to FIGS. 33 and 34, the processor 100 analyzes gaze locationdata 842 to monitor a gaze from a driver. If a gaze location 94 is onthe display 60 for a time 844 that is greater than a threshold time 846,a gaze is registered and the processor 100 generates an auxiliarylocation value 848 at the time 849 the gaze is registered.

Referring to FIGS. 34 and 35, the processor 100 generates (or requestsgeneration of) an auxiliary direction 852 based on the auxiliarylocation value 848. For example, the auxiliary direction 852 is the nextturn. Here, the turn would have been presented later based on a next oneof default location values 812 but is instead presented now because ofthe auxiliary location value 848.

Alternatively, the auxiliary direction 852 is general information suchas “I am working properly, your next turn will be presented in 10seconds.” Here, the next turn is presented in ten seconds when the nextone of the set of default parameter values 812 matches the location ofthe vehicle 10.

In this manner, the computing device 40 responds to the gaze of a driverby increasing the communication of directions 68 to the driver. Forexample, the frequency of communication of directions 68 is increased bycommunicating the auxiliary direction 852 in addition to the defaultdirections 822.

One advantage of this technology is that the computing device 40responds to a driver that is having a new experience and is not calm(e.g., as evidenced by a gaze) by adjusting the default communication toinclude more frequent instructions or general messages. For example, anew experience is driving a route that a driver has not previouslydriven. Alternatively, if a driver is driving a common route, then thelack of gazes reflects the comfort of the driver. In response, thecomputing device 40 includes less frequent instructions (e.g., thedefault directions) and/or limiting messages to only important alerts(e.g., accidents, hazards, changes in traffic).

Referring to FIGS. 36-37, the vehicle system information application 120includes computer-readable instructions that, when executed by theprocessor 100, cause the processor 100 to perform the vehicle systeminformation method 900. According to the vehicle system informationmethod 900, information 68 associated with a vehicle system 20 ispresented when a user gazes for a period of time at a location 902associated with the vehicle system 20.

The information 68 to be communicated by the computing device 40 may bestatic (e.g., fixed or predetermined) or may be dynamic (e.g., maychange over time). The information 68 may be generated and stored to beaccessed by the computing device 40. Alternatively, the computing device40 may generate the information 68 in real-time or request that thevehicle system 20 generate the information 68 in real-time. The location902 associated with the vehicle system 20 includes the indicator 70associated with the vehicle system 20, the controls 22 associated withthe vehicle system 20, and a designated location in the vehicle 10(e.g., on the dashboard 80, steering wheel 82, or center console 86).

At a block 910, the processor 100 analyzes gaze location data 942 (e.g.,horizontal x-axis is time) to monitor a gaze location 94 of a driver. Ifthe gaze location 94 is at the location associated with a vehicle system20 for a time 944 that is greater than a threshold time 946, at a block920, a gaze is registered and, at a block 930, the processor 100communicates the information 68 visually on the display 60 and/oraudibly through the audio system 88 of the vehicle 10.

Using an automatic cruise control (ACC) system 20 as an example of avehicle system 20, the ACC system 20 is activated when the ACC system 20is turned on and when a speed is set using the ACC system 20.

According to one embodiment, the location associated with the ACC system20 is the speedometer 70 and the information 68 includes values on thespeedometer 70 including the current speed, speed limits, and set speedentered by the driver. Alternatively, the information 68 includesgeneral information such as “I am working properly, I am trying to reachyour set speed”.

According to another embodiment, the location associated with the ACCsystem 20 is a set of controls used to provide input (e.g., set speed)to the ACC system 20 and the information 68 includes directions on howto use the set of controls to operate the ACC system 20.

Referring to FIGS. 38-39, the driver context information application 120includes computer-readable instructions that, when executed by theprocessor 100, cause the processor 100 to perform the driver contextinformation method 1000. According to the driver context informationmethod 1000, information 68 is based on a measurement from the sensor 30that represents a driver's context and the information is presented whenthe driver gazes for a period of time at a location associated with avehicle system 20.

The information 68 to be communicated by the computing device 40 may bestatic (e.g., fixed or predetermined) or may be dynamic (e.g., maychange over time). The information 68 may be generated and stored to beaccessed by the computing device 40. Alternatively, the computing device40 may generate the information 68 in real-time or request that thevehicle system 20 generate the information 68 in real-time.

The measurement from the sensor 30 represents the driver's context or isused to identify a context. For example, a context includes the drivingexperience of a driver (e.g., novice/expert, age), environment 50outside the vehicle 10 (e.g., weather, road conditions), environment 50inside the vehicle 10 (e.g., occupancy, temperature), and the status ofthe vehicle systems 20 (e.g., fuel level). Driver experience can bedetermined by facial recognition associated with a driver profile thatincludes an age or experience metric.

At a block 1010, the processor 100 analyzes output data 62, 64, 66 fromthe sensor 30. At a block 1020, the processor 100 identifies orgenerates a context 1022 based on the output data 62, 64, 66.

At a block 1030, the processor 100 identifies or generates contextinformation 1032 that is specific to a vehicle system 20 and the context1022.

Blocks 1040, 1050, 1060 of the driver context information method 1000are similar to blocks 910, 920, 930 of the vehicle system informationmethod 900 and are described with reference to FIG. 37.

At a block 1040, the processor 100 analyzes gaze location data 942 tomonitor a gaze location 94 of a driver. If the gaze location 94 is atthe location 902 associated with a vehicle system 20 for a time 944 thatis greater than a threshold time 946, at a block 1050, a gaze isregistered and, at a block 1060, the processor 100 communicates thecontext information 1032 associated with the vehicle system 20 (i.e.,that which is specific to the vehicle system 20 associated with thelocation 902). The context information 1032 is communicated visually onthe display 60 and/or audibly through the audio system 88 of the vehicle10.

For example, the processor 100 analyzes output data 62 from the sensor30 and, based on the output data 62, determines that the driver is anovice 1022. The processor 100 generates or identifies contextinformation 1032 for the vehicle systems 20 based on the driver being anovice 1022.

Using the ACC system 20 as an example of a vehicle system 20, a location902 associated with the ACC system 20 is the dial 22 for the ACC system20. When the processor 100 registers a gaze at the dial 22 of the ACCsystem 20, the processor 100 accesses the context information 1032 forthe ACC system 20. For example, the context information 1032 includesdirections for operating the ACC system 20 that would be useful to anovice driver. The processor 100 communicates the context information1032 visually on the display 60 and/or audibly through a speaker systemof the vehicle 10.

For example, the processor 100 analyzes an output data 62 from thesensor 30 and, based on the output data 62, determines an occupancy 1022of the vehicle. The processor 100 generates or identifies contextinformation 1032 for the vehicle systems 20 based on the occupancy 1022of the vehicle. Context information 1032 includes that which wouldprovide a more comfortable settings to all passengers based on theoccupancy 1022.

Using a heating ventilation air conditioning (HVAC) system 20 as anexample of a vehicle system 20, a location 902 associated with the HVACsystem 20 is the dial 22 for the HVAC system 20. When the processor 100registers a gaze at the dial 22 of the HVAC system 20, the processor 100accesses the context information 1032 for the HVAC system 20. Forexample, the context information 1032 includes directions for operatingthe back vents of the HVAC system 20 because the occupancy 1022 includespassengers in the back seat of the vehicle 10 (e.g., the car is fullypacked). The processor 100 communicates the context information 1032visually on the display 60 and/or audibly through a speaker system ofthe vehicle 10.

Various embodiments of the present disclosure are disclosed herein. Thedisclosed embodiments are merely examples that may be embodied invarious and alternative forms, and combinations thereof. As used herein,for example, “exemplary,” and similar terms, refer expansively toembodiments that serve as an illustration, specimen, model or pattern.

Variations, modifications, and combinations may be made to theabove-described embodiments without departing from the scope of theclaims. All such variations, modifications, and combinations areincluded herein by the scope of this disclosure and the followingclaims.

1. A method, comprising: facilitating, by a device comprising aprocessor, display of an indicator on a display of a vehicle, theindicator including an indicator area, wherein the indicator isassociated with a vehicle system; comparing, by the device, sensor datato a threshold, wherein the sensor data is associated with the vehiclesystem, wherein the threshold represents a separation between a firststate of the vehicle system and a second state of the vehicle system;comparing, by the device, gaze location data to the indicator area; andreducing, by the device, a prominence of the indicator if: the sensordata is on a normal state side of the threshold; and the gaze locationdata is found in the indicator area.
 2. The method of claim 1, whereinthe indicator includes an image.
 3. The method of claim 1, wherein thefirst state is a normal state and the second state is a critical state.4. The method of claim 1, wherein the sensor data represents a state ofthe vehicle system.
 5. The method of claim 1, wherein reducing theprominence includes reducing a size of the indicator.
 6. The method ofclaim 1, wherein reducing the prominence includes reducing a brightnessof the indicator.
 7. The method of claim 1, wherein reducing theprominence includes changing a type of the indicator.
 8. The method ofclaim 1, wherein reducing the prominence includes changing a position ofthe indicator on the display.
 9. A method, comprising: displaying anindicator on a display of a vehicle, the indicator including anindicator area, wherein the indicator is associated with a vehiclesystem; comparing sensor data to a threshold, wherein the sensor data isassociated with the vehicle system, wherein the threshold represents aseparation between a first state of the vehicle system and a secondstate of the vehicle system; comparing gaze location data to theindicator area; and increasing a prominence of the indicator if: sensordata is on a critical state side of the threshold; and gaze locationdata is not found in the indicator area.
 10. The method of claim 9,wherein the indicator includes an image.
 11. The method of claim 9,wherein the first state is a normal state and the second state is acritical state.
 12. The method of claim 9, wherein the sensor datarepresents a state of the vehicle system.
 13. The method of claim 9,wherein the increasing the prominence includes increasing a size of theindicator.
 14. The method of claim 9, wherein the increasing theprominence includes increasing a brightness of the indicator.
 15. Themethod of claim 9, wherein the increasing the prominence includeschanging a type of the indicator.
 16. The method of claim 9, wherein theincreasing the prominence includes changing a position of the indicatoron the display.
 17. A method, comprising: displaying an indicator on avehicle display, the indicator including an indicator area; calculatinga gaze frequency associated with the first indicator, wherein the gazefrequency is calculated over a period of time and is based on at leastone of: a number of times a gaze location moves into the indicator area;and a time a gaze location spends in the indicator area; and determininga prominence at which to set the indicator based on the gaze frequency.18. The method of claim 17, wherein: the gaze frequency is a first gazefrequency and the period of time is a first period of time; and themethod further comprises: calculating a second gaze frequency over asecond period of time, the second period of time being longer than thefirst period of time; increasing the prominence of the indicator if thefirst gaze frequency is above the second gaze frequency; and decreasingthe prominence of the indicator if the first gaze frequency is below thesecond gaze frequency.
 19. The method of claim 17, wherein prominence isbased on at least one of a size, a brightness, a type, and a position onthe display.
 20. The method of claim 17, wherein: the indicator area isa first indicator area, the gaze frequency is a first gaze frequency andthe period of time is a first period of time; and the method furthercomprises: displaying a second indicator on a vehicle display, thesecond indicator including a second indicator area; calculating a secondgaze frequency associated with the second indicator, wherein the secondgaze frequency is calculated over a second period of time and is basedon at least one of: a number of times a gaze location moves into thesecond indicator area; and a time a gaze location spends in the secondindicator area; and determining the prominence of the second indicatorbased on the second gaze frequency; wherein positions on the display areordered based on prominence and determining the prominence includesdetermining the position of the first indicator and the second indicatorbased on an order of the first gaze frequency and the second gazefrequency.